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The literature on the re-use of learning materials has largely focused on the 
development of materials. This paper explores how re-use can be stimulated after 
learning materials have been developed and made available. We searched for and 
developed guidelines that support staff and/or management most frequently adopt 
in cases of (un)successful re-use of existing digital teaching and learning materials 
and methods by teachers in higher education. In a grounded theory approach, we 
collected existing guidelines from 11 literature studies, and developed new 
guidelines from 19 (mainly Dutch) case studies. Through constant comparison, we 
developed ‘low-level’ guidelines, which were classified into high-level guidelines. 
Five high-level guidelines turned out to be robust when subjected to repeated 
comparison with the case studies. Further, this paper elaborates existing guidelines 
by showing specific elaborations of them in our case studies. 

Keywords: educational innovation; higher education; teacher support; re-use; 
digital learning materials 


Introduction 

There is now an extensive literature on the re-use of learning objects, i.e. resources 
“usually digital and web-based, that can be used and re-used to support learning” 
(Wiley 2000). Up till now, the literature has focused on the development of materials, 
with significant attention being paid to the design of learning objects to optimise their 
re-use. This has led to a debate on the extent to which learning objects should contain 
such diverse elements as context (Quinton 2007), pedagogy (McCormick 2003) and 
learning objectives (Boyle 2003). Learning materials have been classified into various 
hierarchies (Cisco Systems 2003; Wagner 2002; Wiley 2000), and learning object 
design trade-offs have been formulated, e.g. between size and re-usability (South and 
Monson 2000). Attention has also been paid to encouraging re-use by making learning 
materials available through repositories, using metadata (Sonntag 2007). This has led 
to the formulation of principles for good metadata (NISO Press 2004); the develop- 
ment of metadata standards such as Learning Object Metadata (LOM) (Institute of 
Electrical and Electronics Engineers 2002); a discussion on the use of tagging versus 
metadata (Smith 2008); and the fonnulation of a trade-off between learning material 
size and costs of metadata (Wiley 2002). 
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However, much less attention has been paid to the question of how re-use can be 
stimulated after development. In this situation, one or more collections of existing 
learning materials are available, and the question is how can teachers be stimulated to 
re-use these existing materials? This question has become increasingly relevant in 
recent years, as an abundance of learning materials has become available through the 
Internet and through institutional and shared repositories. In this paper we will focus 
on the re-use of existing learning materials in Dutch higher education. 

Re-using existing learning materials in higher education is different from develop- 
ing re-usable learning materials, in several respects. Firstly, it is educational support 
staff and managers rather than developers who support re-use in this situation. 
Secondly, support staff have fewer possibilities than developers to create an optimal 
situation for re-use. They will be confronted with a range of existing materials, vary- 
ing in terms of fit to teachers’ needs, quality of the metadata and appropriateness of 
their size. Thirdly, while developers can stimulate re-use by developing high-quality 
learning materials, support staff focus must be on the teachers, coaching them towards 
successful re-use in their teaching. 

Although this post hoc stimulation of re-use has received less attention than the 
development of learning materials, several guidelines for it can be found in the litera- 
ture. An example is peer review (Campbell 2003; Lim 2007; Metros, Bennett, and 
Diaz 2001), which was shown to resolve teachers’ anxiety about the quality of the 
learning materials (Littlejohn 2003a). 

The aim of our study is to bring together the most important guidelines from the 
literature and develop additional guidelines from existing case studies. We develop a 
theory for supporting the re-use of existing learning materials, which is both 
‘grounded’ in the data (Charmaz 2006) and ‘humble’ in the sense of Cobb et al. (2003, 
10), the theory being “concerned with domain-specific [support] processes, ... 
accountable to the activity of design”, and “providing] detailed guidance in organis- 
ing [support]”. Although these guidelines may be matched onto stages of Rogers’ 
(2003) ‘grand theory’ of the innovation process, this is not the focus of our paper. 
Most of the research for our study was conducted in a six-month study for SURF, the 
Dutch foundation for ICT in Higher Education (Schoonenboom et al. 2007). 

Our main research question is: 

What guidelines do support staff and/or management most frequently follow in successful 
and unsuccessful cases of re-use of existing digital teaching and learning materials and 
methods by teachers in higher education? How can these guidelines be classified? 

In this research question, following a guideline can be done either explicitly or 
implicitly. Some guidelines were mentioned by participants as guidelines they had 
followed, others were derived from information on cases of re-use, and participants 
may not have been aware of them. We looked at successful and unsuccessful cases of 
re-use of existing learning materials, as we would expect that useful guidelines should 
contribute to the success of re-use (and ignoring them may lead to unsuccessful re- 
use). Our research aims to discover the guidelines most frequently followed. The more 
often a guideline is encountered in successful cases, the more likely it is that its 
success is not a coincidence, and the more useful it will be to educational support staff. 
Guidelines are classified not only to arrive at a more manageable presentation, but 
also to arrive at more general guidelines to be followed, and to inform practice in 
several ways. 
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Our criteria for including learning materials and methods is whether they can be 
used by individual teachers in their own teaching. Examples include materials such as 
study tasks, courses, pictures or theses, or methods such as working in small virtual 
groups, concept mapping or the use of webcams. Outside the scope of this research 
lies, for example, the institution-wide deployment of competence-based learning, or 
the development of a master degree at a distance. 


Methodology 

Design 

Guidelines were derived through two methods. Firstly, we searched for guidelines in 
a sample of relevant literature. These guidelines were not treated as ‘proven important 
guidelines’, but as input for the formulation of guidelines. Secondly, we derived 
guidelines from a sample of case studies of successful and unsuccessful re-use of 
existing learning materials. We then applied a grounded theory approach (Charmaz 
2006), in which we derived general guidelines from the literature and case studies 
samples together, using constant comparison. Finally, the resulting guidelines were 
compared with the recent literature. 


Sampling from practice 

We followed a multi-stage purposeful sampling strategy (Onwuegbuzie and Leech 
2007). Our first sample contained case studies of re-use of existing digital learning 
materials and methods in Dutch higher education. Following the general principles of 
Robert Stake (1995, 2006), we selected those cases from which we expected to learn 
most. We used the following criteria for selecting case studies: (1) we knew the cases; 
(2) they were recent; (3) descriptions with some detail were available; (4) we knew 
something interesting was done on re-use of existing materials; (5) something had 
been learned and/or was assumed by the participants on success or fail factors with re- 
use; and (6) which together led to a variety of guidelines. On the basis of the first five 
criteria, we selected a first set of case studies. 

Since it turned out that the guidelines derived from these case studies were too 
limited in scope, we then used theoretical sampling (Glaser and Strauss 1967) to 
increase the variety of the guidelines, to ensure they covered the whole innovation 
process and to find more evidence and counterevidence for our guidelines-in-devel- 
opment. We added a second sample of case studies from an existing collection, 
which had been brought together in our earlier study Learning objects in practice 
(Schoonenboom et al. 2006). In this study, carried out between 2004 and 2006, we 
identified and contributed to solutions for important problems for those working with 
learning objects in Dutch higher education. In this study, a set of six Dutch case 
studies had been collected, which met the following criteria: (1) they were related to 
digital learning materials; (2) they had successfully applied for funding with the 
Dutch national SURF organisation or the Dutch Digital University; (3 ) they had been 
finished successfully; and (4) the results were used in practice. This second sample 
had in the original study already been supplemented by two other case studies of 
successful practice and two running experiments, in order to increase diversity and to 
increase the validity of the discovered guidelines. 

The literature that we used as data for developing guidelines was also taken from 
our study Learning objects in practice (Schoonenboom et al. 2006). At the start of this 
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project in 2004, four learning objects experts, each with four or more years of experi- 
ence (both at a national and an international level) conducted a literature review in the 
problems of working with digital learning materials. 

A description of the 19 case studies and 11 literature studies used in the analyses 
can be found in Schoonenboom et al. (2007). 


Procedure 

The procedure consisted of four steps. Figure 1 shows the results of each step, which 
in the text are indicated in bold. 


Step 1. Identification of ‘actions’ in case studies 

Detailed descriptions of the case studies were either collected or prepared by the 
researchers. Each description included a description of the success or lack of success 
of the practice. Within each example of practice, ‘actions’ were identified, which 
seemed to have yielded or contributed to its success or lack of success. 


The practice: 

The department of Psychology of the Erasmus University in Rotterdam, The Netherlands, has 
set up a repository which contains all relevant educational materials of the Psychology 
program (articles, chapters of books, lectures, animations, video clips, presentations). 
Materials are collected by a small team. 

Description of success or lack of success: 

In the first half year, 4500 resources were collected; in a few years, the collection had grown 
to include 7500 resources, and the project leader claims that users are enthusiastic. 

Short descriptions of actions: 

(1) The repository is organised such as to support the department’s central pedagogy of 
problem-based learning. 

(2) Students can get a view on materials that might be of interest for their specific learning 
goals from different perspectives, such as per problem, course, or program. 

(3) Students can add ratings to materials, write notes or a review and make these visible to 
others. 

Low-level guidelines: 

Ad 1 . Within courses, materials and methods should support a more comprehensive 
pedagogical approach or educational ambition (high-level guideline: 1). 

Ad 2. To the extent that teachers and students themselves have to search or browse for 
learning materials, one search or browse attempt should yield enough, but not too much 
materials which are sensibly ordered (high-level guideline: 2). 

Ad 3. Enable the addition of annotations by users, which can give teachers and students an 
impression of the quality of the materials and methods in practice (high-level guideline: 4). 


Figure 1. Summary of one practice, showing the elements of the derivation of guidelines: 
description of success, short descriptions of actions, low-level guidelines and high-level 
guidelines; numbers of high-level guidelines refer to their number in Figure 2. 
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Step 2. Formulation of low-level guidelines 

For each action, a short description was created. The literature sample was scanned 
for existing guidelines, and short descriptions of these guidelines were prepared. From 
the short descriptions of both the existing guidelines in the literature and the practice 
actions, low-level guidelines were derived. The derivation of the low-level guidelines 
was an iterative process, using repeated comparison (Charmaz 2006; Glaser and 
Strauss 1967). First, a set of guidelines was formulated, and as many short descrip- 
tions as possible were grouped under these guidelines. Reflection on the remaining 
descriptions led to the formulation of new guidelines and so on, until each short 
description was grouped under one low-level guideline. Then, the literature and the 
detailed descriptions of the case studies were analysed again, looking for additional 
instances of actions that would fit under the low-level guidelines. 


Step 3. Formulation of high-level guidelines 

Finally, the low-level guidelines were categorised into high-level guidelines, such as 
‘Provide easy access to the materials and methods’ in Figure 2. The same procedure 
of repeated comparison was used as in step 2. 


Step 4. Testing the validity of the guidelines 

The resulting high-level guidelines were tested for robustness. We performed a split- 
half test, in which we randomly divided the literature and practice cases into two. The 
split-half test showed that guidelines derived from 10 or more cases were robust in 
that they could be derived from either half of the cases. These five guidelines were 
kept, and the other non-robust guidelines were removed. Finally, we checked a recent 
three-volume work on learning objects for remarks on these guidelines (Harman and 
Koohang 2007; Koohang and Harman 2007a, 2007b). 


Validity 

Throughout our study, we had to deal with a variety of validity threats. Consequently, 
we used a variety of strategies to deal with these threats. 

• Case studies were divided among the researchers, and each researcher analysed 
those practices with which he or she was most familiar (Stake 1995, 2006). 
Most of the case studies we knew very well and over a long period, thereby 
reducing the risk of misinterpretations (Becker 1970). 

• IVe used multiple data sources’, we used multiple cases, which were multiple in 
kind, including both descriptions of case studies and literature studies, and by 
multiple authors. This reduced the risk of biases, resulting from studying one 
specific case using a specific type of resource by one specific researcher. 

• Descriptions of case studies had been validated by participants. This applied to 
all samples. 

• To ensure that the guidelines would be robust enough to capture a variety of 
situations, we monitored the guidelines at any point in our research project, to 
see if they provided enough variation. When it turned out that the guidelines of 
our first sample did not explicitly refer to digital learning materials, we included 
a second sample of case studies and literature studies. 
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• The descriptions and analyses of the six most demanding, initially selected case 
studies were either done by two researchers, or by one researcher and checked 
by another. 

• Within each individual practice, we devoted one question in our analysis 
template to a systematic search for additional evidence and for rival explana- 
tions of the presumed relation between action and success (or failure) . Evidence 
often included signs of growing and sustained use. Sometimes, the presumed 
relation was backed by evidence from the literature. 

• With each guideline that we had derived from an individual practice, we 
checked its occurrence in all other case studies. Furthermore, we checked 
whether guidelines found in one case study were contradicted in other case 
studies, but we found no examples of this. 

• Reformulations in the guidelines were thoroughly discussed between 
researchers. 


Outcomes 

Figure 2 shows the five high-level guidelines that we derived. After each guideline, 
the number of cases on which it is based is indicated in brackets. 

Guideline 1 is mostly relevant to managers who at the beginning of the process 
are considering re-use of existing learning materials. Guidelines 2-5 become rele- 
vant once the decision for re-use has been taken and is mostly relevant to support 
staff. 

The sections below each present one high-level guideline, with an explanation of 
how it was derived from its underlying low-level guidelines. A document is available 
in Dutch that shows for all guidelines the whole derivation process from detailed 
descriptions to actions to low-level guidelines to high-level guidelines (Schoonenboom 
et al. 2007). 


1. Develop an appropriate vision on the incipient re-use of the materials and 
methods within the programme, department or institution 

From the beginning an appropriate vision of re-use should be developed by the 
teacher(s), department or institution involved. Our study showed that the following 
elements of this vision, which had been identified in the literature, occur frequently in 
the case studies: 


(1) Develop an appropriate vision on the incipient re-use of the materials within the 
programme, department or institution (N=14). 

(2) Provide easy access to the materials (N=17). 

(3) Provide materials that fit well the teachers’ own needs and way of teaching (N=14). 

(4) Provide materials of good quality, and enable teachers and students to judge the quality 
of materials and methods (N=10). 

(5) Enable necessary adaptations of the materials and methods to the context (N=18). 


Figure 2. The five high-level guidelines for supporting the re-use of existing digital learning 
materials and methods; numbers indicate the number of the 30 case studies and literature stud- 
ies on which each item is based. 
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(1) The materials should have a demonstrable added value for teaching (Rogers 
2003), which is visible to the teachers and managers involved. Examples 
include materials that can be taken into account in students’ assessment and 
materials on which use feedback can be provided by either teacher or 
students. 

(2) Re-use of materials should be considered a gradual process of small changes. 
Teachers are more inclined to apply small innovations than to change the 
whole curriculum (Rogers 2003). 

(3) Re-use of materials should be viewed as a way of improving education, not as 
a way of reducing costs. In fact, costs may go up (Pennings et al. 2005). Our 
case studies show that institutions should consider costs for updating the mate- 
rials (Littlejohn, Jung, and Broumley 2003). Also, materials or methods that 
might be provided for free during the course of a project might require a 
license after the project has finished. 

New in our study is that it shows examples in practice of visions that may stimulate 
re-use. We found that this vision might be a big or a small idea. In one practice, a 
repository was established in support of the ‘big’ idea of problem-based learning. 
Existing learning materials were collected that together support the whole cycle of 
problem-based learning in the department involved. In another practice, the ‘small’ 
idea of visual learning was prominent, and learning materials and methods were 
collected that could contribute to this idea. Although earlier literature has identified 
ideas such as problem-based learning as a basis for developing new learning materials 
(many examples in Koohang and Harman 2007a; Wiley 2003), we found that this is 
also relevant in working with existing learning materials. 

2. Provide easy access to the materials and methods 

If it is too difficult for teachers to get access to the digital learning materials and meth- 
ods, they will not use them. This point has received much attention in the literature on 
learning object development. As a first solution to this problem, the importance of 
good quality metadata has been emphasised, so that users find relevant, and neither 
too few nor too many learning materials (Koper 2003; Sonntag 2007). A second solu- 
tion is the use of ‘tailored wrappers’, information on the local context of use, which is 
added to existing learning materials (Quinton 2007). 

These two solutions show up in our case studies, but with two elaborations. First, 
two case studies show that it can be useful not to provide access to all learning mate- 
rials of an available repository, but instead to make a selection of materials that are 
deemed relevant to the teachers involved. Second, one practice made use of tailored 
wrappers which contained: infonnation on which colleagues had already used these 
materials; infonnation on where one could get help; own metadata with familiar local 
terms which was added on top of the existing metadata. 

3. Provide materials and methods that are a good fit to teachers ’ own needs and 
way of teaching 

If materials and methods are a good fit to teachers’ own needs and ways of teaching, 
then it will be easy for teachers to use them (Rogers 2003). On the other hand, learning 
materials cannot easily be used if, for example, the cultural gap with the teacher’s own 
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practice is too wide (Littlejohn 2003b), or each piece is too large, containing too much 
material that is not useful (Downes 2002). 

In line with the literature, our case studies show that if the conditions of use of the 
target institution are too different from those where the materials were developed, then 
re-use will not be possible. A good fit also implies that teachers should be able to use 
the materials selectively, only the elements they deem to be useful. Student needs 
must be considered too; for example, materials that fit the students’ future profes- 
sional practice can be particularly useful. Materials that are a poor fit include those 
that are too complex for the level at which they are intended to be used, or those that 
clash significantly with aspects of the pedagogy in use, or if they are built upon an 
‘alien’ pedagogy. 


4. Provide materials of good quality, and enable teachers and students to judge the 
quality of materials 

Learning materials should be of good quality. At the level of the repository, too much 
variation between learning materials is considered a problem (Downes 2002). Teach- 
ers and students should be enabled to judge their quality (Littlejohn 2003a); peer 
review is seen as a very useful method (Metros, Bennett, and Diaz 2001). 

In line with the literature, we found that the collection of materials in one repos- 
itory should be consistent, and with respect to the domain involved the materials 
should be as complete as possible, and of equal quality. For students, it is important 
that multimedia and interactive materials are provided (Ravenscroft, Boyle, and 
Jones 2004). Regarding judgement of quality by teachers and students, we found in 
the cases that we studied the use of peer review, in which peer teachers tested the 
usability of materials before use in the institution. In one practice, peer review was 
seen as especially important for materials on the web because of the unclear status 
of these materials. The second form of judgement in our case studies was rating and 
annotating by users to inform their peers on how to use the learning material 
involved (Sonntag 2007). 

In addition to what is known from the literature, we found in one example of prac- 
tice a clear limit to differences in the size of learning materials. Here, teachers did not 
appreciate that the results of one search could include materials as diverse as study 
tasks and pictures. In examining peer review, we found that teachers rely partly on 
whether they know the person who developed the materials and the status that this 
person has. 

5. Enable necessary adaptations of the materials and methods to the context 

T eachers should be able to use the materials and methods in their teaching. An important 
element of this is that they should be able to adapt the materials and methods to their 
own teaching context. Several factors might hinder use. As a necessary condition, 
learning materials should ‘run’ in the institution’s virtual learning environment. Also, 
copyright problems concerning the re-use of the materials should be solved (Campbell 
2003). Teachers must be provided with the necessary tools and support. Lack of user- 
friendly tools with which teachers can re-use learning objects is considered a problem 
(McNaught 2003). Also, time is an important constraining factor (Littlejohn, Jung, and 
Broumley 2003), and enough time should be provided for teachers to evaluate and test 
materials. 
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In line with these findings in the literature, we found in our research that the user- 
friendliness of the materials and the occurrence of technical problems were also 
constraining factors. If adaptation takes too much time, teachers will not use the mate- 
rials. We also found the need for specific forms of support. Support should be 
provided to users at several levels of expertise, for both students, managers, beginning 
teachers and experienced teachers. Explaining the pedagogical possibilities of the 
materials helps, as this may not be clear from the materials themselves. Collective 
support should be complemented by individual support. 

In addition to what was found in the literature, our case studies showed that use of 
the materials is stimulated by removing two important practical hindrances to re-use 
by enabling teachers and students to work with the materials from home and by 
making the necessary software freely available. One additional form of support in the 
case studies was to provide support to beginners by letting them join more experienced 
colleagues. 


Conclusions 

This paper has contributed to the discussion on re-use of learning materials in several 
ways. It has brought together existing answers and developed new answers to ques- 
tions that had perhaps been solved for the development of learning materials from 
scratch, but not for situations in which teachers and their supporters are working with 
existing learning materials. Our case studies have shown that also in this situation, an 
overarching educational idea or vision is required. Furthermore, although the existing 
materials may not all be optimal, access can be optimised by making a relevant selec- 
tion, by adding own metadata on top of existing metadata and by bringing down the 
range of different levels of aggregation of learning materials that teachers can find in 
one search. 

Our study has grounded existing guidelines and provided an overview by showing 
which of these are important in practice, in that they occur often in our cases. It has 
specified existing guidelines, in that it has shown some specific elaborations of these 
guidelines in practice. 

Finally, our paper has demonstrated a method for deriving guidelines from case 
studies, testing their robustness and combining these with findings in the literature. 
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